Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015–20): a comparative analysis

There has been a surge of interest in artificial intelligence and machine learning (AI/ML)-based medical devices. However, it is poorly understood how which AI/ML-based devices have approved the USA Europe. We searched governmental non-governmental databases to identify 222 240 The number increased substantially since 2015, with many being for use radiology. few were qualified as high-risk Of 124 commonly Europe, 80 first One possible reason approval Europe before might be potentially relatively less rigorous evaluation substantial highlight need ensure regulation these Currently, there no specific regulatory pathway or recommend more transparency on are regulated enable improve public trust, efficacy, safety, quality A comprehensive, publicly accessible database device details Conformité Européene (CE)-marked US Food Drug Administration needed. In medicine, aim patient care by uncovering new insights from big data generated an individual collective experience patients.1Hwang TJ Kesselheim AS Vokinger KN Lifecycle intelligence- learning-based software medicine.JAMA. 2019; 3222285Crossref Scopus (23) Google Scholar AI can defined science engineering making intelligent machines, especially computer programmes.2McCarthy J What intelligence?.http://jmc.stanford.edu/articles/whatisai/whatisai.pdfDate: Nov 12, 2007Date accessed: October 1, 2020Google still complete definition AI3Hutson M Glossary: intelligence, so words.Science. 2017; 357: 19Crossref PubMed (11) include different techniques, such ML, statistical methods, expert systems, that primarily rely decision rules. Furthermore, techniques overlap, ML methods.4Bzdok D Altman N Krzywinski Statistics versus learning.Nat Methods. 2018; 15: 233-234Crossref (221) this Health Policy, we (FDA) system capacity learn training task tracking performance measures.5US AdministrationProposed framework modifications intelligence/machine (SaMD).https://www.fda.gov/media/122535/downloadDate: 2019Date Although early systems generally based rules, research focuses methods.6Yu K-H Beam AL Kohane IS Artificial healthcare.Nat Biomed Eng. 2: 719-731Crossref (337) Therefore, purpose study only included using between 2015 2020 refer them methods. When intended diagnose, treat, prevent health problems, under Food, Drug, Cosmetic Act (in USA) Council Directive 93/42/EEC EU countries) Therapeutic Products Switzerland). FDA provides real-world examples devices, imaging uses algorithms give diagnostic information skin cancer smart electrocardiogram estimates probability heart attack.5US their handled differently (table). Instead, risks centralised clears through three pathways: premarket (most stringent review devices), de-novo (for low moderate-risk 510(k) pathway.1Hwang For simplicity, clearance devices.TableOverview EuropeUSAEurope*We member states, EFTA countries, Turkey.Regulatory agencyOrganisationFDAAccredited private Notified Bodies; manufacturer's self-responsibility (low risk) devices; mutual recognition States, TurkeyCentralised decentralisedCentralisedDecentralisedRegulatory pathwaySpecific devicesNoneNone; general requirements performance, reliability; clinical studies assess vary across BodiesPremarket approvalMost category (class III); must provide valid scientific evidence non-clinical showing safety effectivenessNA510(k) pathwayFor class I, II III not indicated; submitters compare one similar legally marketed dataNADe-novo reviewFor I controls alone, special controls, reasonable assurance effectiveness useNAApproval pathwayType approvalApproval FDACE markPublic access documentsYesVery limited availabilityCE=Conformité Européenne. EFTA=European Free Trade Association. FDA=US Administration. NA=not applicable.* Turkey. Open table tab CE=Conformité applicable. Unlike USA, agency lowest risk (ie, I), manufacturer sole responsibiliaty products comply regulations without undergoing process. Medical high classes (IIa, IIb, III) so-called Bodies—ie, organisations accredited do conformity assessment issue Européenne (CE) mark.7Vokinger Mühlematter UJ Becker Boss Reutter MA Szucs TD und der medizin.https://jusletter.weblaw.ch/juslissues/2017/903/artificial-intellige_da49225588.html__ONCE&login=falseDate: Aug 28, 2017Date December 26, same approach applies in-vitro (IVD).8SwissmedicGuide devices.https://www.swissmedic.ch/swissmedic/en/home/medical-devices/regulation-of-medical-devices/medical-device-regulation_online-guide.htmlDate: 6, European Association States (Liechtenstein, Iceland, Norway, Switzerland), Turkey concluded treaties regarding assessments devices.8SwissmedicGuide countries. Each countries recognise CE mark issued Bodies other Germany is, example, automatically also recognised France, England, select Body any bodies, academia, industry.9Wang F Casalino LP Khullar Deep Learning medicine-promise, progress, challenges.JAMA Intern Med. 179: 293-294Crossref (55) Scholar, 10Beam Big Data care.JAMA. 319: 1317-1318Crossref (430) announced 2019 was reviewing regulate devices.5US Previous analysed, among things, chances ethical implications.1Hwang 11Char DS Shah NH Magnus Implementing care—addressing challenges.N Engl 378: 981-983Crossref (270) Nonetheless, our knowledge comprehensive analysis Attempts widely hampered absence available register (by contrast USA), confidentiality submitted regulators, decentralised nature decisions.12Hwang Sokolov E Franklin JM Comparison rates issues reporting trial outcomes Union United States: cohort study.BMJ. 2016; 353i3323Crossref (34) 13Kramer DB Xu S How does perform union? systematic review.PLoS 2012; 9e1001276Crossref (73) To assist current discussions legislative reforms aimed analyse marked 2020. all eligible following characteristics collected: name, manufacturer, date, use, pathway.14Topol EJ High-performance medicine: convergence human intelligence.Nat 25: 44-56Crossref (966) Additionally, extracted specialty applied logic CE-marked devices.15US AdministrationDevice classification panels.https://www.fda.gov/medical-devices/classify-your-medical-device/device-classification-panelsDate: 31, 2018Date commercial Orbis (Netherlands)16Bureau van DijkOrbis.https://www.bvdinfo.com/en-gb/our-products/data/international/orbisDate: 2020Date Eikon Refinitiv (New York, NY, USA):17EikonRefinitiv.https://www.refinitiv.com/en/products/eikon-trading-softwareDate: origin country case residencies than country, selected residence where developed) size company (based Small Business Administration's states small publisher businesses those operations US$38·5 million annual revenue fewer 500 employees).18US AdministrationTable standards.https://www.sba.gov/document/support--table-size-standardsDate: non-registered manufacturers, consulted manufacturers' website categorised companies large basis employees listed website. Descriptive statistics done variables trends over time. Pearson χ2 tests Yates' correction used relationship categorical (medical specialty, origin, health-care professional application vs direct use) Post-hoc analyses further combination responsible significant (p<0·05). adjusted analyses, did multivariable linear regression both time differences jurisdictions, significance model. user professionals use), size, (USA only). Sensitivity presence outliers CE-marking years continuous, instead categorical, robustness All R (version 3.6.1). Our search identified 15 525 sources interest. final 202 (91%) searches 150 (63%) (figure appendix pp 3–37). AI/ML-devices nine 13 2016, 32 2017, 67 2018, 77 2019. Between Jan 1 March 2020, 24 2). 204 (92%) pathway, (7%) 3 (1%) 3). 129 (585) radiology, followed 40 (58%) cardiovascular 21 (9%) neurological proportion radiology six 16 47 189 (85%) FDA-approved professionals, whereas 33 (15%) patient. 172 (77%) 50 (23%) companies. majority manufacturers during (126 [57%] Israel (16 [7%]), Sweden (ten [5%]), England (nine [4%]), France (eight [4%]). Significant associations observed (p=0·011). show molecular genetics considerably higher expected null hypothesis (p=0·008). No devices' (p=1·00), (p=0·16), (p=0·88; 38–41). 27 26 55 100 19 85 (35%) 95 (40%) IIa, 28 (12%) 11 (5%) IVD two III. (8%) 126 (53%) 30 (13%) hospital (10%) 12 ophthalmic (4%) pathological Other specialties gastroenterology urology (6 [3%]), chemistry (four [2%]), anaesthesiology (three [1%]), microbiology (two obstetrics gynaecology dental [1%]) haematology [0·4%]). radiological 10 2016 (37%), 2017 (58%), decrease 2018 (51%), 59 (59%) 196 (82%) 44 (18%) use. 193 (80%) manufactured (20%) at approval. (n=221) (p<0·001), (p<0·001). England-based (p<0·001) Israel-based IIa (p=0·02) hypothesis. Higher counts (p=0·04), (p=0·01), haematological IIb (p=0·01). (p=0·13) (appendix 42–47). (27%) 462 AI/ML-medical median difference 18 days marking 137 (116 [94%]), 117 (5 [4%], figure 4). 98 but 116 USA. Among most (80 [65%]), 9 86 (69%) developed 38 (31%) (38 [31%] devices) larger non-commonly (9 [9%] [8%] devices). Multivariate showed 166 results 48–53). continuously 2), (240 comparison (222 amount underestimated because Only high-risk—ie, (3 (2 [1%]). (129 [58%] [53%] Europe) specialties. contributing factor grow disproportionate rate when compared trained readers.19Hosny Parmar C Quackenbush Schwartz LH Aerts HJWL radiology.Nat Rev Cancer. 18: 500-510Crossref (590) decline reimbursements forced providers compensate increasing productivity.20McDonald RJ KM Eckel LJ et al.The effects changes utilization technological advancements cross-sectional radiologist workload.Acad Radiol. 2015; 22: 1191-1198Summary Full Text PDF (101) collected routine practice, datasets readily efficient apply; therefore, offering incredibly useful resource discovery. Overall, 2015. It could indication related will continue increase. allow conclusions about whether actually practice. Conversely, experiences, hospitals institutions internally mark;21Pram O Liability law: careful manufacturing yourself.https://www.aerzteblatt.de/archiv/195575/Haftungsrecht-Vorsicht-bei-der-Eigenherstellung-von-MedizinproduktenDate: however, scope analysis. pathway. Two breast detection assistance abnormality 3–20). classified Both support cardiac MRI CT post-processing workflows 21–37). Notably, leniant pathways—eg, MammoScreen (Therapixel, Valbonne, France) support, Cardio (Arterys, San Francisco, CA, part Arterys MICA statistically pathways patient, manufacturers. From perspective, association would applications supporting crucial decisions, including monitoring analytics acute blood loss (eg, Early Bird Bleed Monitoring System (Saranas, Houston, TX, pulmonary nodule scans syngo.CT Lung CAD [Siemens Solutions, Malvern, PA, USA]). genetics. (MUTYH-Associated Polyposis [23andMe, Mountain View, USA]), affects made. (Israel England) assessed explain results. free choose shall marked—ie, (this called forum shopping).22Arznei-telegrammSchrott mit CE-zeichhen—medizinprodukte Europa.https://www.arznei-telegramm.de/html/2012_11/1211089_01.htmlDate: 2012Date apply standards classification.22Arznei-telegrammSchrott period. These make up 56% 52% marked. likely get had mark, 4), consistent findings previous study.12Hwang This strenghtened report highlighting later found unsafe ineffective.12Hwang important note necessary draw evidence-based scarce.12Hwang Commission's (Eudamed2) restricted available) repository market surveillance exchanged national competent authorities Commission.23European CommissionMedical devices—EUDAMED.https://ec.europa.eu/growth/sectors/medical-devices/new-regulations/eudamed_enDate: June 17, 2022, (Eudamed) become fully operational. By database, some registration certificates conformity, summary performance).24European CommissionFact sheet MDR information.https://ec.europa.eu/health/sites/health/files/md_newregulations/docs/transparency_factsheet_en.pdfDate: July believe step understanding evaluate opportunities urge researchers publish summaries produce improved documents offer insufficient Also currently advisable. provided statements vague scarce, makes challenging confine device's capabilities. SnoreSounds (Appian Medical, Severna Park, MD, described statement “software ... indicated aid snoring patients”.25US Department & Human ServicesK150102 (SnoreSounds).https://www.accessdata.fda.gov/cdrh_docs/pdf15/K150102.pdfDate: Oct 7, 2015Date January 13, 2021Google Because scarce statement, search. via Factiva, [that] employs neural network sleep breathing sounds caused airway collapse apnea”.26iCrowdNewswireAppian developed/validated (93% accurate) diagnose apnea snore recordings smartphone.https://icrowdnewswire.com/2016/07/29/appian-medical-developedvalidated-93-accurate-software-can-diagnose-sleep-apnea-using-snore-recordings-smartphone/Date: 29, 2016Date helpful if AI/ML-based, potential improvements practice27Triantafyllidis AK Tsanas Applications real-life digital interventions: literature.J Med Internet Res. 21e12286Crossref (58) limitations clear.6Yu 28Ching T Himmelstein Beaulieu-Jones BK al.Opportunities obstacles deep biology medicine.J Soc Interface. 1520170387Crossref (577) 29Hwang Sachs RE Evaluating rules drug development.JAMA Oncol. 5: 461-462Crossref (2) Transparency allowing agencies, researchers, successes failures reflects responsibility regulators manufacturers.29Hwang Detecting challenging, lack terms associated AI/ML websites news articles. discrepancy led lower detected study.30Benjamens Dhunnoo P Meskó B state intelligence-based algorithms: online database.NPJ Digit 2020; 3: 118Crossref (63) limitations. First, unified AI/ML. broader lead Second, always determine (even after contacting manufacturers). Third, exhaustive strong incentives sell possible,12Hwang represent major Finally, FDA's approach,15US classifications possible. specialty. fraction FDA. importance More advisable summaries)